Feature selection for intrusion detection system

Intrusion detection is an important task for network operators in today’s Internet. Traditional network intrusion detection systems rely on either specialized signatures of previously seen attacks, or on labeled traffic datasets that are expensive and difficult to reproduce for user-profiling to hunt out network attacks. Machine learning methods could be used in this area since they could get knowledge from signatures or as normal-operation profiles. However, there is usually a large volume of data in intrusion detection systems, for both features and instances. Feature selection can be used to optimize the classifiers used to identify attacks by removing redundant or irrelevant features while improving the quality. In this thesis, six feature selection algorithms are developed, and their application to intrusion detection is evaluated. They are: Cascading Fuzzy C Means Clustering and C4.5 Decision Tree Classification Algorithm, New Evidence Accumulation Ensemble with Hierarchical Clustering Algorithm, Modified Mutual Information-based Feature Selection Algorithm, Mutual Information-based Feature Grouping Algorithm, Feature Grouping by Agglomerative Hierarchical Clustering Algorithm, and Online Streaming Feature Selection Algorithm. All algorithms are evaluated on the KDD 99 dataset, the most widely used data set for the evaluation of anomaly detection methods, and are compared with other algorithms. The potential application of these algorithms beyond intrusion detection is also examined and discussed.